Electronic device using external device and operation

ABSTRACT

An electronic device includes a sensor; a memory storing instructions; and a processor configured to execute the instructions to: estimate a field of view (FOV) of a user by using another sensor included in a wireless earphone; estimate a FOV of the electronic device by using the sensor; compare the estimated FOV of the user with the estimated FOV of the electronic device; determine whether the user gazes at a screen of the electronic device based on the comparison result; recognize a gaze of the user based on determining that the user gazes at the screen of the electronic device; and perform a specified function based on the gaze of the user.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation application of InternationalApplication No. PCT/KR2022/013387, filed on Sep. 6, 2022, which claimspriority to Korean Patent Application No. 10-2021-0140501, filed on Oct.20, 2021, in the Korean Intellectual Property Office, the disclosures ofwhich are incorporated by reference herein in their entireties.

BACKGROUND 1. Field

The disclosure relates to an electronic device using an external deviceand a method thereof.

2. Description of Related Art

User interfaces of electronic devices are being diversified with theadvancement of technology. Although input from software keys or hardwarekeys has been mainly used as user interfaces, user interfaces usingsensors are being diversified as electronic devices can include varioussensors. For example, an electronic device may identify the iris of theuser by using a camera to thereby unlock itself or execute a designatedapplication.

As electronic devices can include various configurations and the numberof functions that can be performed increases, power consumption is alsorapidly increasing. In particular, since a camera may consume a largeamount of power, it may be difficult to utilize a function using thecamera for a long time. Various embodiments of the disclosure mayprovide a method for replacing a function using a camera with a functionusing a sensor other than a camera, and an electronic device using thesame.

SUMMARY

According to an aspect of the disclosure, an electronic device includes:a sensor; a memory storing instructions; and a processor configured toexecute the instructions to: estimate a field of view (FOV) of a user byusing another sensor included in a wireless earphone; estimate a FOV ofthe electronic device by using the sensor; compare the estimated FOV ofthe user with the estimated FOV of the electronic device; determinewhether the user gazes at a screen of the electronic device based on thecomparison result; recognize a gaze of the user based on determiningthat the user gazes at the screen of the electronic device; and performa specified function based on the gaze of the user.

According to an aspect of the disclosure, a wireless earphone includes:a sensor; a memory storing instructions; and a processor configured toexecute the instructions to: estimate a posture of the wireless earphonebased on data obtained using the sensor; estimate a posture of a head ofa user based on the posture of the wireless earphone; and estimate afield of view (FOV) of the user based on the posture of the head of theuser.

According to an aspect of the disclosure, an operation method of anelectronic device includes: estimating a field of view (FOV) of a userby using a first sensor included in a wireless earphone; estimating aFOV of the electronic device by using a second sensor included in theelectronic device; comparing the estimated FOV of the user with theestimated FOV of the electronic device; determining whether the usergazes at a screen of the electronic device based on the comparing;recognizing a gaze of the user based on determining that the user gazesat the screen of the electronic device; and performing a specifiedfunction of the electronic device based on the gaze of the user.

According to an aspect of the disclosure, an operation method of awireless earphone, the operation method includes: estimating a postureof the wireless earphone based on data obtained using a sensor;estimating a posture of a head of a user based on the posture of thewireless earphone; and estimating a field of view (FOV) of the userbased on the posture of the head of the user.

According to one or more embodiments of the disclosure, an electronicdevice may identify whether the user is looking at the electronic devicewithout using a camera.

According to one or more embodiments of the disclosure, an electronicdevice may confirm face recognition and/or gaze recognition withoutusing a camera or using a camera only when necessary, so that power(e.g., current) consumption of the electronic device can be reduced.When the consumption of current is reduced, the lifespan of the batterycan be extended and the amount of heat generated by the electronicdevice can be reduced.

According to one or more embodiments of the disclosure, an electronicdevice may always run the face recognition and/or gaze recognitionfunction if the current consumption for confirming the face recognitionand/or gaze recognition is small. When the face recognition and/or gazerecognition function is always run, the initial driving time required torun an application using the face recognition and/or gaze recognitionfunction may be reduced.

According to various embodiments of the disclosure, privacy can beprotected by not using a camera.

BRIEF DESCRIPTION OF THE DRAWINGS

The above and other aspects, features, and advantages of certainembodiments of the present disclosure will be more apparent from thefollowing description taken in conjunction with the accompanyingdrawings, in which:

FIG. 1 is a block diagram of an electronic device in a networkenvironment according to various embodiments;

FIG. 2 illustrates an electronic device connectable to an externaldevice according to various embodiments;

FIG. 3 is an internal block diagram of a wireless earphone according tovarious embodiments;

FIG. 4 is an internal block diagram of an electronic device according tovarious embodiments;

FIG. 5A is a diagram depicting a user gazing at the screen of anelectronic device according to an embodiment, and FIG. 5B is a diagramdepicting a FOV of the user gazing at the electronic device in FIG. 5A;

FIG. 6 is a flowchart of the electronic device according to variousembodiments;

FIG. 7A is a flowchart for estimating a user's FOV according to variousembodiments;

FIG. 7B shows an example of estimating the posture of the wirelessearphone using a gyro sensor and an acceleration sensor;

FIG. 7C shows an example of a vector indicating the posture of thewireless earphone and a vector indicating the posture of the user's headwhen the user wears the wireless earphone;

FIG. 7D shows various examples of a FOV of the user;

FIG. 8A illustrates coordinate axes for the posture of an electronicdevice according to various embodiments, and FIG. 8B illustrates a FOVof the electronic device according to an embodiment;

FIG. 9 is a diagram illustrating a reference coordinate systemrepresenting a gaze vector and a reference coordinate systemrepresenting a vector indicating a screen direction of the electronicdevice according to various embodiments;

FIG. 10A illustrates a method of measuring a distance by using a UWBsignal according to various embodiments;

FIG. 10B illustrates a diagram in which the electronic device measures adistance to the wireless earphones by using a UWB signal according tovarious embodiments;

FIGS. 10C to 10F illustrate the user's gaze direction represented usingthe distances and directions of a first wireless earphone and a secondwireless earphone measured by a UWB communication module of theelectronic device;

DETAILED DESCRIPTION

FIG. 1 is a block diagram illustrating an electronic device 101 in anetwork environment 100 according to certain embodiments. Referring toFIG. 1 , the electronic device 101 in the network environment 100 maycommunicate with an electronic device 102 via a first network 198 (e.g.,a short-range wireless communication network), or at least one of anelectronic device 104 or a server 108 via a second network 199 (e.g., along-range wireless communication network). According to an embodiment,the electronic device 101 may communicate with the electronic device 104via the server 108. According to an embodiment, the electronic device101 may include a processor 120, memory 130, an input module 150, asound output module 155, a display module 160, an audio module 170, asensor module 176, an interface 177, a connecting terminal 178, a hapticmodule 179, a camera module 180, a power management module 188, abattery 189, a communication module 190, a subscriber identificationmodule (SIM) 196, or an antenna module 197. In some embodiments, atleast one of the components (e.g., the connecting terminal 178) may beomitted from the electronic device 101, or one or more other componentsmay be added in the electronic device 101. In some embodiments, some ofthe components (e.g., the sensor module 176, the camera module 180, orthe antenna module 197) may be implemented as a single component (e.g.,the display module 160).

The processor 120 may execute, for example, software (e.g., a program140) to control at least one other component (e.g., a hardware orsoftware component) of the electronic device 101 coupled with theprocessor 120, and may perform various data processing or computation.According to one embodiment, as at least part of the data processing orcomputation, the processor 120 may store a command or data received fromanother component (e.g., the sensor module 176 or the communicationmodule 190) in volatile memory 132, process the command or the datastored in the volatile memory 132, and store resulting data innon-volatile memory 134. According to an embodiment, the processor 120may include a main processor 121 (e.g., a central processing unit (CPU)or an application processor (AP)), or an auxiliary processor 123 (e.g.,a graphics processing unit (GPU), a neural processing unit (NPU), animage signal processor (ISP), a sensor hub processor, or a communicationprocessor (CP)) that is operable independently from, or in conjunctionwith, the main processor 121. For example, when the electronic device101 includes the main processor 121 and the auxiliary processor 123, theauxiliary processor 123 may be adapted to consume less power than themain processor 121, or to be specific to a specified function. Theauxiliary processor 123 may be implemented as separate from, or as partof the main processor 121.

The auxiliary processor 123 may control at least some of functions orstates related to at least one component (e.g., the display module 160,the sensor module 176, or the communication module 190) among thecomponents of the electronic device 101, instead of the main processor121 while the main processor 121 is in an inactive (e.g., sleep) state,or together with the main processor 121 while the main processor 121 isin an active state (e.g., executing an application). According to anembodiment, the auxiliary processor 123 (e.g., an image signal processoror a communication processor) may be implemented as part of anothercomponent (e.g., the camera module 180 or the communication module 190)functionally related to the auxiliary processor 123. According to anembodiment, the auxiliary processor 123 (e.g., the neural processingunit) may include a hardware structure specified for artificialintelligence model processing. An artificial intelligence model may begenerated by machine learning. Such learning may be performed, e.g., bythe electronic device 101 where the artificial intelligence is performedor via a separate server (e.g., the server 108). Learning algorithms mayinclude, but are not limited to, e.g., supervised learning, unsupervisedlearning, semi-supervised learning, or reinforcement learning. Theartificial intelligence model may include a plurality of artificialneural network layers. The artificial neural network may be a deepneural network (DNN), a convolutional neural network (CNN), a recurrentneural network (RNN), a restricted Boltzmann machine (RBM), a deepbelief network (DBN), a bidirectional recurrent deep neural network(BRDNN), deep Q-network or a combination of two or more thereof but isnot limited thereto. The artificial intelligence model may, additionallyor alternatively, include a software structure other than the hardwarestructure.

The memory 130 may store various data used by at least one component(e.g., the processor 120 or the sensor module 176) of the electronicdevice 101. The various data may include, for example, software (e.g.,the program 140) and input data or output data for a command relatedthereto. The memory 130 may include the volatile memory 132 or thenon-volatile memory 134.

The program 140 may be stored in the memory 130 as software, and mayinclude, for example, an operating system (OS) 142, middleware 144, oran application 146.

The input module 150 may receive a command or data to be used by anothercomponent (e.g., the processor 120) of the electronic device 101, fromthe outside (e.g., a user) of the electronic device 101. The inputmodule 150 may include, for example, a microphone, a mouse, a keyboard,a key (e.g., a button), or a digital pen (e.g., a stylus pen).

The sound output module 155 may output sound signals to the outside ofthe electronic device 101. The sound output module 155 may include, forexample, a speaker or a receiver. The speaker may be used for generalpurposes, such as playing multimedia or playing record. The receiver maybe used for receiving incoming calls. According to an embodiment, thereceiver may be implemented as separate from, or as part of the speaker.

The display module 160 may visually provide information to the outside(e.g., a user) of the electronic device 101. The display module 160 mayinclude, for example, a display, a hologram device, or a projector andcontrol circuitry to control a corresponding one of the display,hologram device, and projector. According to an embodiment, the displaymodule 160 may include a touch sensor adapted to detect a touch, or apressure sensor adapted to measure the intensity of force incurred bythe touch.

The audio module 170 may convert a sound into an electrical signal andvice versa. According to an embodiment, the audio module 170 may obtainthe sound via the input module 150, or output the sound via the soundoutput module 155 or a headphone of an external electronic device (e.g.,an electronic device 102) directly (e.g., wiredly) or wirelessly coupledwith the electronic device 101.

The sensor module 176 may detect an operational state (e.g., power ortemperature) of the electronic device 101 or an environmental state(e.g., a state of a user) external to the electronic device 101, andthen generate an electrical signal or data value corresponding to thedetected state. According to an embodiment, the sensor module 176 mayinclude, for example, a gesture sensor, a gyro sensor, an atmosphericpressure sensor, a magnetic sensor, an acceleration sensor, a gripsensor, a proximity sensor, a color sensor, an infrared (IR) sensor, abiometric sensor, a temperature sensor, a humidity sensor, or anilluminance sensor.

The interface 177 may support one or more specified protocols to be usedfor the electronic device 101 to be coupled with the external electronicdevice (e.g., the electronic device 102) directly (e.g., wiredly) orwirelessly. According to an embodiment, the interface 177 may include,for example, a high definition multimedia interface (HDMI), a universalserial bus (USB) interface, a secure digital (SD) card interface, or anaudio interface.

A connecting terminal 178 may include a connector via which theelectronic device 101 may be physically connected with the externalelectronic device (e.g., the electronic device 102). According to anembodiment, the connecting terminal 178 may include, for example, a HDMIconnector, a USB connector, an SD card connector, or an audio connector(e.g., a headphone connector).

The haptic module 179 may convert an electrical signal into a mechanicalstimulus (e.g., a vibration or a movement) or electrical stimulus whichmay be recognized by a user via his tactile sensation or kinestheticsensation. According to an embodiment, the haptic module 179 mayinclude, for example, a motor, a piezoelectric element, or an electricstimulator.

The camera module 180 may capture a still image or moving images.According to an embodiment, the camera module 180 may include one ormore lenses, image sensors, image signal processors, or flashes.

The power management module 188 may manage power supplied to theelectronic device 101. According to one embodiment, the power managementmodule 188 may be implemented as at least part of, for example, a powermanagement integrated circuit (PMIC).

The battery 189 may supply power to at least one component of theelectronic device 101. According to an embodiment, the battery 189 mayinclude, for example, a primary cell which is not rechargeable, asecondary cell which is rechargeable, or a fuel cell.

The communication module 190 may support establishing a direct (e.g.,wired) communication channel or a wireless communication channel betweenthe electronic device 101 and the external electronic device (e.g., theelectronic device 102, the electronic device 104, or the server 108) andperforming communication via the established communication channel. Thecommunication module 190 may include one or more communicationprocessors that are operable independently from the processor 120 (e.g.,the application processor (AP)) and supports a direct (e.g., wired)communication or a wireless communication. According to an embodiment,the communication module 190 may include a wireless communication module192 (e.g., a cellular communication module, a short-range wirelesscommunication module, or a global navigation satellite system (GNSS)communication module) or a wired communication module 194 (e.g., a localarea network (LAN) communication module or a power line communication(PLC) module). A corresponding one of these communication modules maycommunicate with the external electronic device via the first network198 (e.g., a short-range communication network, such as Bluetooth™,wireless-fidelity (Wi-Fi) direct, or infrared data association (IrDA))or the second network 199 (e.g., a long-range communication network,such as a legacy cellular network, a 5th generation (5G) network, anext-generation communication network, the Internet, or a computernetwork (e.g., LAN or wide area network (WAN)). These various types ofcommunication modules may be implemented as a single component (e.g., asingle chip), or may be implemented as multi components (e.g., multichips) separate from each other. The wireless communication module 192may identify and authenticate the electronic device 101 in acommunication network, such as the first network 198 or the secondnetwork 199, using subscriber information (e.g., international mobilesubscriber identity (IMSI)) stored in the subscriber identificationmodule 196.

The wireless communication module 192 may support a 5G network, after a4th generation (4G) network, and next-generation communicationtechnology, e.g., new radio (NR) access technology. The NR accesstechnology may support enhanced mobile broadband (eMBB), massive machinetype communications (mMTC), or ultra-reliable and low-latencycommunications (URLLC). The wireless communication module 192 maysupport a high-frequency band (e.g., the mmWave band) to achieve, e.g.,a high data transmission rate. The wireless communication module 192 maysupport various technologies for securing performance on ahigh-frequency band, such as, e.g., beamforming, massive multiple-inputand multiple-output (massive MIMO), full dimensional MIMO (FD-MIMO),array antenna, analog beam-forming, or large scale antenna. The wirelesscommunication module 192 may support various requirements specified inthe electronic device 101, an external electronic device (e.g., theelectronic device 104), or a network system (e.g., the second network199). According to an embodiment, the wireless communication module 192may support a peak data rate (e.g., 20 Gbps or more) for implementingeMBB, loss coverage (e.g., 164 dB or less) for implementing mMTC, orU-plane latency (e.g., 0.5 ms or less for each of downlink (DL) anduplink (UL), or a round trip of 1 ms or less) for implementing URLLC.

The antenna module 197 may transmit or receive a signal or power to orfrom the outside (e.g., the external electronic device) of theelectronic device 101. According to an embodiment, the antenna module197 may include an antenna including a radiating element composed of aconductive material or a conductive pattern formed in or on a substrate(e.g., a printed circuit board (PCB)). According to an embodiment, theantenna module 197 may include a plurality of antennas (e.g., arrayantennas). In such a case, at least one antenna appropriate for acommunication scheme used in the communication network, such as thefirst network 198 or the second network 199, may be selected, forexample, by the communication module 190 (e.g., the wirelesscommunication module 192) from the plurality of antennas. The signal orthe power may then be transmitted or received between the communicationmodule 190 and the external electronic device via the selected at leastone antenna. According to an embodiment, another component (e.g., aradio frequency integrated circuit (RFIC)) other than the radiatingelement may be additionally formed as part of the antenna module 197.

According to certain embodiments, the antenna module 197 may form ammWave antenna module. According to an embodiment, the mmWave antennamodule may include a printed circuit board, an RFIC disposed on a firstsurface (e.g., the bottom surface) of the PCB, or adjacent to the firstsurface and capable of supporting a designated high-frequency band(e.g., the mmWave band), and a plurality of antennas (e.g., arrayantennas) disposed on a second surface (e.g., the top or a side surface)of the PCB, or adjacent to the second surface and capable oftransmitting or receiving signals of the designated high-frequency band.

At least some of the above-described components may be coupled mutuallyand communicate signals (e.g., commands or data) therebetween via aninter-peripheral communication scheme (e.g., a bus, general purposeinput and output (GPIO), serial peripheral interface (SPI), or mobileindustry processor interface (MIPI)).

According to an embodiment, commands or data may be transmitted orreceived between the electronic device 101 and the external electronicdevice 104 via the server 108 coupled with the second network 199. Eachof the electronic devices 102 or 104 may be a device of a same type as,or a different type, from the electronic device 101. According to anembodiment, all or some of operations to be executed at the electronicdevice 101 may be executed at one or more of the external electronicdevices 102, 104, or 108. For example, if the electronic device 101should perform a function or a service automatically, or in response toa request from a user or another device, the electronic device 101,instead of, or in addition to, executing the function or the service,may request the one or more external electronic devices to perform atleast part of the function or the service. The one or more externalelectronic devices receiving the request may perform the at least partof the function or the service requested, or an additional function oran additional service related to the request, and transfer an outcome ofthe performing to the electronic device 101. The electronic device 101may provide the outcome, with or without further processing of theoutcome, as at least part of a reply to the request. To that end, acloud computing, distributed computing, mobile edge computing (MEC), orclient-server computing technology may be used, for example. Theelectronic device 101 may provide ultra low-latency services using,e.g., distributed computing or mobile edge computing. In anotherembodiment, the external electronic device 104 may include anInternet-of-things (IoT) device. The server 108 may be an intelligentserver using machine learning and/or a neural network. According to anembodiment, the external electronic device 104 or the server 108 may beincluded in the second network 199. The electronic device 101 may beapplied to intelligent services (e.g., smart home, smart city, smartcar, or healthcare) based on 5G communication technology or IoT-relatedtechnology.

FIG. 2 illustrates an electronic device connectable to an externaldevice according to various embodiments.

According to an embodiment, the electronic device 210 (e.g., electronicdevice 101 in FIG. 1 ) may be connected to wireless earphones 220 (e.g.,electronic device 102 in FIG. 1 ) being an external device. The wirelessearphones 220 may be composed of a first wireless earphone 220-1 and asecond wireless earphone 220-2. The first wireless earphone 220-1 andthe second wireless earphone 220-2 may be worn on both ears of the user,and the right and left sides may be distinguished. One of the firstwireless earphone 220-1 and the second wireless earphone 220-2 may playa primary role, and the other may play a secondary role. The wirelessearphone playing the primary role may be directly connected to theelectronic device 210. With reference to FIG. 2 , the first wirelessearphone 220-1 may play a primary role and may be directly connected tothe electronic device 210. The wireless earphone playing the secondaryrole may be synchronized with the wireless earphone playing the primaryrole by transmitting and/or receiving signals (and/or data) to and/orfrom the wireless earphone playing the primary role. With reference toFIG. 2 , the second wireless earphone 220-2 may be a wireless earphoneplaying a secondary role.

According to an embodiment, the first wireless earphone 220-1 may play asecondary role, and the second wireless earphone 220-2 may play aprimary role. The roles of the wireless earphones may be switched witheach other.

In various embodiments of the disclosure, wireless earphones aredescribed as an external device, but a device that can be mounted on auser's head and include a sensor may be an external device described invarious embodiments of the disclosure. For example, wired earphones,headphones, and/or smart glasses may be an external device.

In the disclosure, wireless earphones are described, but a wearabledevice that can be worn on the head (e.g., headband including at leastsome of a sensor, a processor, and a communication module) may also beapplied.

FIG. 3 is an internal block diagram of a wireless earphone according tovarious embodiments.

With reference to FIG. 3 , the wireless earphone (e.g., first wirelessearphone 220-1 in FIG. 2 ) may include a processor 310, a communicationmodule (communication interface) 320, a memory 330, and/or a sensormodule (sensor) 340.

According to an embodiment, the processor 310 may control thecommunication module 320 to process data and/or control signals receivedfrom the electronic device (e.g., electronic device 210 in FIG. 2 ). Forexample, when the processor 310 receives a control signal for volumeadjustment by controlling the communication module 320, it may adjustthe volume of a speaker. According to an embodiment, the processor 310may process data measured using the sensor module 340. For example, theprocessor 310 may process data measured using the sensor module 340 toestimate the posture of the wireless earphone 220-1. The processor 310may estimate the posture of the wireless earphone 220-1 in considerationof the direction (e.g., left or right) of the wireless earphone. Theposture of the wireless earphone 220-1 may be represented in coordinateaxes. According to an embodiment, when the wireless earphone 220-1 playsa primary role, the processor 310 may estimate the posture of thewireless earphone 220-1.

According to an embodiment, the communication module 320 may connect thewireless earphone 220-1 to the electronic device 210. The wirelessearphone 220-1 and the electronic device 210 may be connected throughBluetooth communication, for example, and the communication module 320may support Bluetooth communication and/or another short-rangecommunication standard and/or protocol. When the wireless earphone 220-1plays a secondary role, the communication module 320 may eavesdrop onthe communication between the wireless earphone playing a primary role(e.g., second wireless earphone 220-2) and the electronic device 210 soas to be synchronized with the wireless earphone 220-2 playing theprimary role. Alternatively or additionally, the communication module320 may transmit the state of the wireless earphone 220-1 to thewireless earphone 220-2 playing the primary role. The communicationmodule 320 may transmit the posture of the wireless earphone 220-1estimated by the processor 310 to the electronic device 210.

According to an embodiment, the memory 330 may be configured to storethe state of the wireless earphone 220-1. The processor 310 may storethe currently set state of the wireless earphone 220-1 in the memory330.

According to an embodiment, the sensor module 340 may include anacceleration sensor 340-1 and a gyro sensor 340-2. The processor 310 mayprocess data received from the acceleration sensor 340-1 to estimate theacceleration of the wireless earphone 220-1. The processor 310 mayprocess data received from the gyro sensor 340-2 to measure therotational angular velocity of the wireless earphone 220-1 in one ormore directions. The processor 310 may estimate the posture of thewireless earphone 220-1 by using the measured acceleration androtational angular velocity of the wireless earphone 220-1.

FIG. 4 is an internal block diagram of an electronic device according tovarious embodiments.

With reference to FIG. 4 , the electronic device (e.g., electronicdevice 101 in FIG. 1 or electronic device 210 in FIG. 2 ) may include afirst processor 410, a second processor 420, a display module 430, acamera module 440, a sensor module (sensor) 450, a communication module(communication interface) 460, and/or a memory 490.

According to an embodiment, the first processor 410 may be a higherperformance processor than the second processor 420. The first processor410 may be, for example, an application processor. The first processor410 may execute an application (e.g., applications 146 in FIG. 1 ). Thefirst processor 410 may be operably, functionally, and/or electricallyconnected to the second processor 420, the display module 430, and/orthe camera module 440.

According to an embodiment, the second processor 420 may be a microcontroller unit (MCU). The second processor 420 may be a processorcapable of driving simple logics with low power. According to anembodiment, the second processor 420 may be operably, functionally,and/or electrically connected to the sensor module 450 and/or thecommunication module 460. When the electronic device 210 includes thefirst processor 410, the second processor 420 may be an optionalcomponent. If the second processor 420 is not included in the electronicdevice 210, those components (e.g., sensor module 450 and/orcommunication module 460) having been described as being connected tothe second processor 420 may be operably, functionally, and/orelectrically connected to the first processor 410.

According to an embodiment, the display module 430 may be a component tovisually provide data to the user. The display module 430 may visuallydisplay data provided by a running application.

According to an embodiment, the camera module 440 may capture a stillimage and/or a moving image. For example, the camera module 440 mayinclude red, green, blue (RGB) camera, a time-of-flight (TOF) camera, adepth camera, and/or a light detection and ranging (LiDAR) scanner. Thefirst processor 410 may recognize the user's face by using the cameramodule 440 and recognize the eyeball (or gaze) of the user.

According to an embodiment, the sensor module 450 may include anacceleration sensor 450-1, a gyro sensor 450-2, a proximity sensor450-3, and/or an infrared sensor 450-4.

According to an embodiment, the second processor 420 may measure theacceleration of the electronic device 210 by processing data receivedfrom the acceleration sensor 450-1.

According to an embodiment, the second processor 420 may measure therotational angular velocity in one or more directions of the electronicdevice 210 by processing data received from the gyro sensor 450-2. Thesecond processor 420 may estimate the posture of the electronic device210 by using the measured acceleration and rotational angular velocityof the electronic device 210.

According to an embodiment, the configuration of the sensor module 450for estimating the posture of the electronic device 210 may be notlimited to the acceleration sensor 450-1 and/or the gyro sensor 450-2.For example, the sensor module 450 may further include a geomagneticsensor and/or a barometric pressure sensor. The geomagnetic sensor maybe used to determine the orientation of the electronic device 210, andthe barometric pressure sensor may be used to measure the altitude ofthe electronic device 210.

According to an embodiment, the second processor 420 may determinewhether the user's face is within a preset range from the electronicdevice 210 by using the proximity sensor 450-3. When the infrared (IR)sensor 450-4 (e.g., IR light-emitting diode (LED)) and the camera module440 (e.g., iris scanner) are used, the electronic device 210 mayrecognize the user's iris.

According to another embodiment, the electronic device 210 may include amillimeter wave (mmWave) communication module (e.g., 60 GHz WirelessFidelity (Wi-Fi)), and may determine whether the user's face is within apreset range from the electronic device 210 by using the mmWavecommunication module.

According to an embodiment, the communication module 460 may include ashort-range communication module 470 and/or a cellular communicationmodule 480.

According to an embodiment, the short-range communication module 470 mayinclude, for example, a Bluetooth (BT)/Bluetooth low energy (BLE) module470-1, a Wi-Fi module 470-2, an ultra-wideband (UWB) module 470-3,and/or a near field communication (NFC) module 470-4.

According to an embodiment, the BT/BLE module 470-1 may supportBluetooth communication so that the electronic device 210 may connect toan external device by using Bluetooth communication.

According to an embodiment, the Wi-Fi module 470-2 may support Wi-Ficommunication. The electronic device 210 may connect to an externaldevice by using Wi-Fi communication.

According to an embodiment, the UWB module 470-3 may support UWBcommunication. The UWB module 470-3 may include at least two UWBantennas (e.g., antenna module 197 in FIG. 1 ). The electronic device210 may identify the location of an external device based on the roundtrip time (RTT) and angle-of-arrival (AOA) of radio frequency (RF)signals received through the UWB antennas from the external device. Forexample, after transmitting a distance measurement request message tothe external device through the UWB antennas, the electronic device 210may identify the round-trip time (RTT) required for receiving a responsemessage as to the distance measurement request from the external device,and may identify the TOF, which is the time required for a radio wave toreach the external device after being emitted from the electronic device210, based on the RTT. Through the TOF, the electronic device 210 mayidentify the relative distance between the electronic device 210 and theexternal device. Alternatively or additionally, the electronic device210 may include at least three UWB antennas, the electronic device 210may identify the angle of arrival (AOA) of the RF signals received fromthe external device based on the phase difference between the RF signalsreceived by the UWB antennas from the external device and the physicaldistance between the spaced-apart UWB antennas, and may identify thedirection in which the external device is located based on the angle ofarrival (AOA). The electronic device 210 may measure the relativeposition between the electronic device 210 and the external device(e.g., distance to and/or angle with the external device) based on theTOF and AOA identified using UWB communication.

According to an embodiment, the NFC 470-4 may support short-rangewireless communication. The NFC 470-4 may use a frequency of about 13.56MHz to exchange data at a short distance within about 10 cm withoutcontact, for example.

According to an embodiment, the cellular communication module 480 mayinclude, for example, Long Term Evolution (LTE)/Fifth Generation (5G)module 480-1. The LTE/5G module 480-1 may support an LTE communicationmethod and/or a 5G communication method. The electronic device 210 maycommunicate with a base station by using LTE communication/5Gcommunication. According to an embodiment, the cellular communicationmodule 480 may also support Second Generation (2G) and/or ThirdGeneration (3G) communication schemes.

According to an embodiment, the memory 490 may temporarily and/ornon-temporarily store information processed by the processor (e.g.,first processor 410 and/or second processor 420). For example, thememory 490 may store location information of the external device basedon the relative distance between the electronic device 210 and theexternal device and/or location information of the electronic device210.

According to an embodiment, the memory 490 may store instructions thatmay be executed by the first processor 410 and/or the second processor420, as described in various embodiments of the disclosure. The firstprocessor 410 and/or the second processor 420 may operate by reading theinstructions from the memory 490.

FIG. 5A is a diagram depicting a user gazing at the screen of anelectronic device according to an embodiment, and FIG. 5B is a diagramdepicting a FOV of the user gazing at the electronic device in FIG. 5A.

With reference to FIG. 5A, the user 500 may use the electronic device(e.g., electronic device 210 in FIG. 2 ) while wearing the wirelessearphone (e.g., first wireless earphone 220-1 in FIG. 2 ). For example,the user 500 may make a video call with another user by using thewireless earphone 220-1 and the electronic device 210. The user 500 mayview a counterpart user's image (e.g., the image of the another user) byusing the electronic device 210 while listening to a counterpart user'svoice by using the wireless earphone 220-1. As another example, the user500 may view images by using the wireless earphone 220-1 and theelectronic device 210. The user 500 may listen to audio through thewireless earphone 220-1 while viewing images by using the electronicdevice 210.

According to an embodiment, for the user 500 to view the screen (ordisplay) of the electronic device 210, the degree of inclination of theelectronic device 210 and the degree of inclination of the user must bewithin a specific range with respect to the direction of gravity 505. InFIG. 5A, the degree of inclination of the electronic device with respectto the direction of gravity 505 is represented by a first vector 510 andthe degree of inclination of the user is represented by a second vector520.

In various embodiments of the disclosure, fields of view (FOVs) may bedefined respectively for the user and the electronic device. The FOV ofthe electronic device may mean a range in which a face and/or gaze canbe recognized through a camera and/or UWB, and the FOV of the user maymean a range that the user can observe with their eyes. In FIG. 5A, thevector serving as a reference for the FOV of the electronic device isindicated by a third vector 515, and the vector serving as a referencefor the user's FOV is indicated by a fourth vector 525.

According to an embodiment, the third vector 515 may be a vectorperpendicular to the screen of the electronic device 210. The anglebetween the third vector 515 and the direction of gravity 505 may be atilt angle. According to an embodiment, the fourth vector 525 may bereferred to as a gaze vector in the direction of the user's gaze.

According to an embodiment, the degree of inclination of the electronicdevice and the user may be represented by the first vector 510 and thesecond vector 520 with respect to the direction of gravity 505, but mayalso be represented by the third vector 515 and the fourth vector 525.

According to an embodiment, the FOV of the user may be subdivided anddistinguished. With reference to FIG. 5B, a general FOV 540 may be arange that the user may observe with the eyes, and may be about 180degrees. In various embodiments of the disclosure, the range that theuser can see with an intention may be defined as a user FOV 530. Forexample, when the electronic device 210 is located within the user FOV530, the user may focus on data displayed on the screen of theelectronic device 210. The user FOV 530 may be further subdivided asshown in FIG. 7D below.

FIG. 6 is a flowchart of the electronic device according to variousembodiments.

According to an embodiment, at operation 610, the electronic device(e.g., electronic device 210 in FIG. 2 or FIG. 4 ) may estimate the FOVof the user (e.g., user FOV 530 in FIG. 5B). The electronic device 210may receive, through the communication module (e.g., communicationmodule 460 in FIG. 4 ), data obtained by a sensor (e.g., sensor module340 in FIG. 3 ) included in the wireless earphone (e.g., first wirelessearphone 220-1 in FIG. 2 or 3 ), and estimate the FOV of the user. Theelectronic device 210 may receive, through the communication module, theFOV of the user estimated directly by the wireless earphone using dataobtained from the sensor. A method for the electronic device 210 or thewireless earphone to estimate the user's FOV is described in detailbelow with reference to FIGS. 7A to 7D.

According to an embodiment, at operation 620, the electronic device 210may estimate the FOV of the electronic device 210. As described in FIG.5A, the FOV of the electronic device 210 may be a range in which theuser's face and/or gaze can be recognized using a camera (e.g., cameramodule 440 in FIG. 4 ) included in the electronic device 210. When theelectronic device 210 supports UWB communication by using the UWB (e.g.,UWB 460-3 in FIG. 4 ), the FOV of the electronic device 210 may be arange in which the user's face and/or gaze can be recognized using thewireless earphone and UWB communication. A method by which theelectronic device 210 estimates the FOV of the electronic device 210 isdescribed in detail below with reference to FIGS. 8A and 8B.

According to an embodiment, at operation 630, the electronic device 210may compare the estimated FOV of the user with the estimated FOV of theelectronic device 210. The electronic device 210 may compare the vector(e.g., third vector 515 in FIG. 5A) that is a reference of the user'sFOV and the vector (e.g., fourth vector 525 in FIG. 5A) that is areference of the FOV of the electronic device.

According to an embodiment, at operation 640, the electronic device 210may determine whether the user gazes at the screen of the electronicdevice 210 based on the comparison result. When the estimated FOV of theuser and the estimated FOV of the electronic device 210 fall within apreset range, the electronic device 210 may determine that the user isgazing at the screen of the electronic device 210. The electronic device210 may compare the vector (e.g., third vector 515) that is a referenceof the user's FOV with the vector (e.g., fourth vector 525) that is areference of the FOV of the electronic device 210 to determine whetherthe two vectors are parallel or face each other with respect to thedirection of gravity (e.g., direction of gravity 505 in FIG. 5A). Whenthe difference between the two vectors is maintained within a specificrange in consideration of various variables such as shaking of theelectronic device 210, movement of the user's face, and/or movement ofthe user's eyes, the electronic device 210 may determine that the usergazes at the screen of the electronic device 210.

According to an embodiment, upon determining that the user gazes at thescreen of the electronic device 210, at operation 650, the electronicdevice 210 may recognize the user's gaze. When it is determined that theuser gazes at the screen of the electronic device 210, the electronicdevice 210 may determine whether it is possible to recognize the user'sgaze by using at least some of the camera, the sensor, and the UWBcommunication module to recognize the user's gaze. If it is determinedthat the user does not gaze at the screen of the electronic device 210,the electronic device 210 may proceed to operation 610 again (e.g., 640NO). A method by which the electronic device 210 determines whether theuser gazes at the screen of the electronic device 210 is described indetail below with reference to FIGS. 10A to 10F.

According to an embodiment, at operation 660, the electronic device 210may perform a specified function of the electronic device 210 based onthe recognized user's gaze. When the user's gaze is recognized, theelectronic device 210 may perform a specified function of the electronicdevice 210 based on the recognized user's gaze.

According to an embodiment, the electronic device 210 may determinewhether the user's gaze is recognized only when the user gazes at thescreen of the electronic device 210, and when the user's gaze isrecognized, the electronic device 210 may perform a preset function ofthe electronic device 210 only through settings without usermanipulation. As a result, the electronic device 210 may not have todetermine whether the user's gaze is recognized, thereby reducing theconsumption of current, and the user may not need to perform an actionto execute a specified function (e.g., action to run an application).

FIG. 7A is a flowchart for estimating a user's FOV according to variousembodiments.

According to an embodiment, at operation 710, the electronic device(e.g., electronic device 210 in FIG. 2 or FIG. 4 ) may utilize dataobtained through a sensor included in the wireless earphone (e.g., firstwireless earphone 220-1 in FIG. 2 or FIG. 3 ) to estimate the posture ofthe wireless earphone. For example, the wireless earphone 220-1 mayinclude an acceleration sensor (e.g., acceleration sensor 340-1 in FIG.3 ) and a gyro sensor (e.g., gyro sensor 340-2 in FIG. 3 ). Theprocessor (e.g., processor 310 in FIG. 3 ) of the wireless earphone220-1 may use the acceleration sensor 340-1 to estimate the degree ofinclination of the sensor axis of the wireless earphone 220-1 withrespect to the direction of gravity. The processor 310 of the wirelessearphone 220-1 may estimate the relative change in posture of thewireless earphone 220-1 by using the gyro sensor 340-2. According to anembodiment, the data measured using the gyro sensor 340-2 may beaccurate for a short time, but may have a drift error when beingintegrated to obtain a value for a long time. The data measured usingthe acceleration sensor 340-1 may be inaccurate due to noise in a shortmoment, but the value for a long time may be accurate. By using thecomplementary characteristics of the gyro sensor 340-2 and theacceleration sensor 340-1, a relatively accurate posture of the wirelessearphone 220-1 (or, the vector indicating the posture of the wirelessearphone 220-1) can be estimated.

FIG. 7B shows an example of estimating the posture of the wirelessearphone using a gyro sensor and an acceleration sensor.

With reference to FIG. 7B, the data measured using the accelerationsensor 340-1 may include noise when there is no movement. Data measuredusing the acceleration sensor 340-1 may be used to identify a slow andlarge motion by removing noise with a low-pass filter 740. The datameasured using the gyro sensor 340-2 may include a drift error whenthere is no movement. The data measured using the gyro sensor 340-2 maybe used to identify a relatively small rotational motion by removing adrift error with a high-pass filter 750. By summing the data measuredusing the acceleration sensor 340-1 that has passed the low-pass filter740 and the data measured using the gyro sensor 340-2 that has passedthe high-pass filter 750, the posture of the wireless earphone 220-1 maybe estimated with relatively high accuracy. The method shown in FIG. 7Bmay be a method using complementary filters, and extended Kalmanfilters, Mahony filters, and/or Madgwick filters may be used.

Such a method of estimating the posture of the wireless earphone 220-1may be in consideration of the user's situation. For example, when theuser gazes at the screen of the electronic device, as the user hardlymoves the head, the stationary state of the user's head may be detectedby the acceleration sensor 340-1, and the drift may be estimated withdata measured using the gyro sensor 340-2. After the drift is estimatedin this way, the estimated drift can be removed from the data measuredusing the sensor, so that the posture of the wireless earphone 220-1 canbe estimated more accurately.

With reference back to FIG. 7A, according to an embodiment, at operation720, the electronic device 210 may estimate the posture of the user'shead (or, the vector representing the posture of the user's head) byusing the estimated posture of the wireless earphone (or, the vectorrepresenting the posture of the wireless earphone). The vectorrepresenting the posture of the wireless earphone and the vectorrepresenting the posture of the user's head may be not the same. Theposture of the wireless earphone may be different depending on thedirection of the wireless earphone (e.g., right or left).

FIG. 7C shows an example of a vector 740 indicating the posture of thewireless earphone and a vector 750 indicating the posture of the user'shead when the user wears the wireless earphone.

According to an embodiment, when the user wears the wireless earphone,the vector 740 indicating the posture of the wireless earphone and thevector 750 indicating the posture of the user's head may be representedby a constant relationship according to the mechanical characteristicsof the wireless earphone. The electronic device 210 may estimate thevector 750 representing the posture of the user's head by using thevector 740 representing the estimated posture of the wireless earphoneand a constant value.

According to an embodiment, the vector 750 indicating the posture of theuser's head may be represented in quaternions, which is one of theschemes for representing the posture. The reference coordinate systemrepresenting the posture may be the Earth's local coordinate system(navigation frame), the vector 740 representing the posture of thewireless earphone may be represented by q_(E) ^(N), and the vector 750representing the posture of the user's head may be represented by q_(H)^(N). When the user is wearing the wireless earphone, the relationshipbetween the vector 740 representing the posture of the wireless earphoneand the vector 750 representing the posture of the user's head may berepresented by q_(H) ^(E). When represented as described above, thevector 750 representing the posture of the user's head may be expressedas in Equation 1 below.

q _(H) ^(N) =q _(E) ^(N) ·q _(H) ^(E)  [Equation 1]

Here, the relationship q_(H) ^(E) between the vector 740 indicating theposture of the wireless earphone and the vector 750 indicating theposture of the user's head when the user is wearing the wirelessearphone may be a fixed value, and may vary depending on the directionof the wireless earphone.

According to an embodiment, the electronic device 210 may estimate thevector representing the front of the user's gaze by using the vector 750representing the posture of the user's head. If the gaze directionvector 760 expressed in the vector 750 representing the posture of theuser's head is {right arrow over (v)}, the gaze direction vector 760{right arrow over (v)}′ expressed in the reference coordinate system canbe represented as Equation 2.

{right arrow over (v)}′=q _(H) ^(N) ·{right arrow over (v)}·(q _(H)^(N))⁻¹  [Equation 2]

According to an embodiment, at operation 730, the electronic device 210may estimate the user FOV by using the vector 750 indicating theestimated posture of the user's head. The electronic device 210 maydefine a specific angular range with respect to the gaze directionvector 760 ({right arrow over (v)}) obtained at operation 720 as theuser FOV.

FIG. 7D shows various examples of the FOV of the user.

With reference to FIG. 7D, the user can recognize text only when thetext is within about +10 degrees with respect to the gaze directionvector 760. The user FOV that the user can recognize text can be definedas about +10. As another example, the user can recognize a symbol onlywhen the symbol is within about +20 degrees with respect to the gazedirection vector 760, and the user can recognize a color only when thecolor is within about +30 degrees with respect to the gaze directionvector 760. According to an embodiment, the user FOV may be differentlydetermined according to an application or function that the user intendsto use. A plurality of user FOVs may be defined, and different user FOVsmay be used according to applications or functions. For example, whenthe user intends to read an article by using an Internet browser, theuser FOV may be based on about +10 degrees at which the user canrecognize text. When the FOV of the electronic device (e.g., electronicdevice 210 in FIG. 2 or FIG. 4 ) deviates from the user FOV, the screenof the electronic device 210 may be blurred to prevent exposure ofcontents. When the FOV of the electronic device 210 falls within theuser FOV, the screen of the electronic device 210 may be made clear. Asanother example, the user FOV for the user to view a video may bedefined as about +20 degrees. When the FOV of the electronic device 210deviates from the user FOV, the video may be stopped, when the FOV ofthe electronic device 210 falls within the user FOV, the video may beresumed. As another example, the user FOV for a reminder or notificationfunction may be defined as about +30 degrees.

Although the electronic device 210 is described as estimating the user'sFOV in FIG. 7A, the wireless earphone 220-1 may directly estimate theuser's FOV by using data obtained from a sensor. When the wirelessearphone 220-1 directly estimates the user's FOV, the wireless earphone220-1 may transmit the estimated user's FOV to the electronic device210. When the alarm sounds, if the FOV of the electronic device iswithin the user FOV (e.g., +30 degrees), the alarm can be stopped, andthe contents of the reminder can be read or displayed. Furthermore, whenthe FOV of the electronic device falls within a more defined user FOV(e.g., +10 degrees), alarm confirmation (e.g., stopping the alarm) maybe performed.

FIG. 8A illustrates coordinate axes for the posture of an electronicdevice according to an embodiment, and FIG. 8B illustrates a FOV of theelectronic device according to an embodiment.

According to an embodiment, the electronic device (e.g., electronicdevice 210 in FIG. 2 or FIG. 4 ) may use data obtained by using a sensorincluded in the electronic device 210 to determine the posture of theelectronic device 210 (or, the vector indicating the posture of theelectronic device). The method for the electronic device 210 to estimatethe posture of the electronic device 210 by using data acquired througha sensor included in the electronic device 210 may be the same as and/orsimilar to the method at operation 710 in FIG. 7A for the electronicdevice 210 to estimate the posture of the wireless earphone 220-1 (thevector indicating the posture of the wireless earphone 220-1) by usingdata obtained through a sensor included in the wireless earphone 220-1.

According to an embodiment, similarly to the vector representing theposture of the user's head, the vector representing the posture of theelectronic device 210 may also represent the posture of the electronicdevice 210 in the Earth's local coordinate system. With reference toFIG. 8A, in a situation where the user views the screen from the frontof the electronic device 210, the user's gaze vector may coincide withthe −z axis among the coordinate axes for the posture of the electronicdevice 210. Hence, the screen direction vector of the electronic device210 may be calculated by defining the vector d in the screen directionof the electronic device 210 as the −z axis and converting it into areference coordinate system. The vector of the screen direction of theelectronic device 210 can be represented as Equation 3 when expressedalso in quaternions.

{right arrow over (d)}′=q _(M) ^(N) ·{right arrow over (d)}(q _(M)^(N))⁻¹  [Equation 3]

Here q_(M) ^(N) may be a vector for the posture of the electronic devicerepresented in the reference coordinate system.

FIG. 8B shows various examples of the FOV of the electronic device as inFIG. 7D. The FOV of the electronic device 210 may vary according to thelocation of a sensor for estimating the posture of the electronic device210. The range of the FOV of the electronic device 210 capable ofrecognizing the user's face and/or gaze by using the camera 810 includedin the electronic device 210 may be different from the range of the FOVof the electronic device 210 in which the electronic device 210 mayrecognize the user's face and/or gaze by using the wireless earphone andthe UWB communication 820.

FIG. 9 is a diagram illustrating a reference coordinate systemrepresenting a gaze vector and a reference coordinate systemrepresenting a vector indicating a screen direction of the electronicdevice according to various embodiments.

With reference to FIG. 9 , the reference coordinate system 910representing the gaze vector (e.g., fourth vector 525 in FIG. 5A) maynot coincide with the reference coordinate system 920 representing thevector indicating the screen direction of the electronic device (e.g.,third vector 515 in FIG. 5A). However, since the gravitational field iscommon, the axes corresponding to the direction of gravity may coincide,and two axes orthogonal to the direction of gravity may not coincide.

According to an embodiment, the electronic device (e.g., electronicdevice 210 in FIG. 2 or FIG. 4 ) may align the axes that do not match tocompare the directions of the gaze vector 525 and the vector 515indicating the screen direction of the electronic device. That is, theelectronic device 210 may correct angle θ 930 tilted in a horizontalcomponent to 0.

According to an embodiment, before correcting angle θ 930 to 0, theelectronic device 210 may only compare the degrees of inclination withrespect to the direction of gravity. The degree of inclination can berepresented as a ratio between the magnitude of the horizontal componentand the magnitude of the vertical component. In FIG. 9 , the x-axis andy-axis may be a horizontal component, and the z-axis may be a verticalcomponent. In this case, the degree of inclination may be represented asin Equation 4 below.

tilt=a tan(√{square root over (x ² +y ²)},z)  [Eq. 4]

In a case in which the values obtained by calculating the degrees ofinclination of the gaze vector 525 and the vector 515 indicating thescreen direction of the electronic device 210 are expressed respectivelyas tilt_v and tilt_d, when the difference between the two values iswithin the FOV, the electronic device 210 may determine that there is ahigh possibility that the user is viewing the screen of the electronicdevice 210.

According to an embodiment, when there is a high possibility that theuser is looking at the screen of the electronic device 210, theelectronic device 210 may drive a sensor (e.g., camera) for gazerecognition. According to another embodiment, when there is a highpossibility that the user is looking at the screen of the electronicdevice 210, the electronic device 210 may display necessary informationon the screen of the electronic device 210.

According to an embodiment, the electronic device 210 may drive a sensor(e.g., camera) for gaze recognition only when there is a highprobability that the user is looking at the screen of the electronicdevice 210, so that current consumption required to drive the sensor(e.g., camera) for gaze recognition can be reduced.

FIG. 10A illustrates a method of measuring a distance by using a UWBsignal according to various embodiments.

FIG. 10A illustrates a method in which the first electronic device 1001transmits a first data frame (1011), the second electronic device 1003transmits a second data frame including the time when the first dataframe is received and the time when the second electronic devicetransmits the data frame (1013), and the first electronic device 1001calculates the distance between the first electronic device 1001 and thesecond electronic device 1003 by further using the information includedin the received second data frame. The first electronic device 1001 maycalculate the distance to the second electronic device 1003 bysubtracting the time at which the first data frame is transmitted andthe time required for the second electronic device 1003 to receive thefirst data frame and transmit the second data frame from the time atwhich the second data frame is received and dividing the result by 2.

FIG. 10B illustrates a diagram in which the electronic device measures adistance to the wireless earphones by using a UWB signal according tovarious embodiments.

With reference to FIG. 10B, the electronic device (e.g., electronicdevice 210 in FIG. 2 ) may use a UWB signal to measure the distance d1to the first wireless earphone (e.g., first wireless earphone 220-1 inFIG. 2 ) and the distance d2 to the second wireless earphone (e.g.,second wireless earphone 220-2 in FIG. 2 ). The electronic device 210may use a plurality of antennas to measure the direction of the firstwireless earphone and the direction of the second wireless earphonebased on a difference between times at which UWB signals are received.

FIGS. 10C to 10F illustrate the user's gaze direction represented usingthe distances and directions of the first wireless earphone and thesecond wireless earphone measured by the UWB communication module of theelectronic device.

According to an embodiment, the electronic device (e.g., electronicdevice 210 in FIG. 2 ) may measure the distances and directions of thefirst wireless earphone and the second wireless earphone by using UWBsignals, and may convert them into coordinate values for positions. Theelectronic device 210 may estimate the perpendicular bisector of a linesegment connecting the coordinate values for the position of the firstwireless earphone and the coordinate values for the position of thesecond wireless earphone as the user's gaze vector. The electronicdevice 210 may extend the user's gaze vector to determine whether it iswithin a specific region of the electronic device.

With reference to FIGS. 10C and 10D, since the user's gaze vector facesthe electronic device and is within a specific region of the electronicdevice when extended, the electronic device 210 may determine that theuser gazes at the screen of the electronic device. According to anembodiment, the electronic device 210 may further identify the positionof the user's head (or face) by using the screen vector of theelectronic device. FIG. 10C may illustrate the user viewing the screenof the electronic device from the front, and FIG. 10D may illustrate theuser viewing the screen of the electronic device from above. Upondetermining that the user is viewing the screen of the electronic deviceas in FIG. 10C or 10D, the electronic device may perform a presetfunction.

With reference to FIGS. 10E and 10F, the user's gaze vector may not facethe electronic device and may be out of the range of the electronicdevice even if extended. When the coordinate values for the position ofthe first wireless earphone and the coordinate values for the positionof the second wireless earphone are estimated as shown in FIG. 10E or10F, the electronic device 210 may determine that the user does not gazeat the screen of the electronic device.

According to various embodiments of the disclosure, the electronicdevice (e.g., electronic device 210 in FIG. 2 or FIG. 4 ) may releaseand/or set the lock screen of the electronic device by using the faceand/or gaze recognition function described above. The electronic device210 may estimate the angle at which the user gazes at the electronicdevice 210. The electronic device 210 may release and/or set the lockscreen of the electronic device in stages by distinguishing the angle atwhich the user gazes at the electronic device 210. For example, theelectronic device 210 may distinguish four stages for the angle at whichthe user gazes at the electronic device 210: a first stage for the anglewithin about +10 degrees, a second stage for within about +20 degrees, athird stage for within about +30 degrees, and a fourth stage for withinabout +40 degrees. When being determined to be the first stage, theelectronic device 210 may determine that the user gazes directly at theelectronic device 210. When being determined to be the second stage, theelectronic device 210 may determine that the user gazes at theelectronic device 210 from the side. When being determined to be thethird stage, the electronic device 210 may determine that the user isgazing at the electronic device 210 in a state where the user's face isdeviated. When being determined to be the fourth stage, the electronicdevice 210 may determine that the user does not gaze at the electronicdevice 210. The electronic device 210 may prepare a function forreleasing the lock screen when the angle at which the user gazes at theelectronic device 210 goes from the fourth stage to the third stage. Forexample, the electronic device 210 may wake up the processor (e.g., AP)in sleep state, prepare to run the camera, and/or prepare to drive UWB.When the second stage is entered, the electronic device 210 may turn onthe screen and display information on the lock screen. When the firststage is entered, the electronic device 210 may execute an operation andinterface for authentication such as face authentication and/or irisauthentication. When authentication is completed, the electronic device210 may release the lock screen.

As another example, in case that the user is not operating theelectronic device 210, the lock screen may be set when the angle atwhich the user gazes at the electronic device 210 goes from the firststage to the second stage, and the screen may be turned off when theangle goes to the third stage or fourth stage.

In the present disclosure, the angle at which the user gazes at theelectronic device 210 is described as being divided into four stages,but it may be further or less subdivided, and the number of stages mayvary according to the application.

According to various embodiments of the disclosure, the electronicdevice (e.g., electronic device 210 in FIG. 2 or FIG. 4 ) mayautomatically adjust the luminance and/or refresh rate of the display(or screen) of the electronic device by using the face and/or gazerecognition function described above. For example, when the user's gazeis recognized while the screen is turned on (e.g., first stage and/orsecond stage), the electronic device 210 may brighten the screen and/orset a high refresh rate, when the user (or, user's face) deviates fromthe preset FOV (e.g., third stage and/or fourth stage), the electronicdevice 210 may lower the screen brightness and/or set a low refreshrate. As another example, even when it goes from the first stage inwhich the user gazes at the electronic device 210 to the second or thirdstage, the electronic device 210 may maintain the display luminanceregardless of the illuminance value. Even if the user and the display'sgaze angle is increased, since the user can gaze with a glance view, theelectronic device 210 may maintain the luminance of the display in orderto preserve the viewing sensation from the side depending on thedisplay. When the fourth stage is exited, the electronic device 210 maychange the luminance of the display.

According to various embodiments of the disclosure, the electronicdevice (e.g., electronic device 210 in FIG. 2 or FIG. 4 ) may change thedisplayed information by using the face and/or gaze recognition functiondescribed above. When information is displayed on the screen with lowpower such as always on display (AOD), the electronic device 210 maydisplay different information or adjust the amount of displayedinformation according to the direction of the user's head. Uponestimating that the user is not looking at the screen (e.g., fourthstage), only minimal information such as time information may bedisplayed, when the electronic device is recognized within the user'sFOV (e.g., first to third stages), more information may be displayed.

According to an embodiment, the electronic device 210 may not displayany information in the fourth stage, may display general information(e.g., weather) or whether there is a message (e.g., 0 messages) when itgoes to the third or second stage, and may display personalizedinformation (e.g., personal schedule, contact information, healthinformation) in the first stage. Sensitive information such as financialinformation may be not displayed at the notification stage and can bechecked after authentication. Information that can be displayed in eachstage can be configured by the user.

An electronic device according to various embodiments of the disclosuremay include: a sensor module, a memory, and a processor, wherein thememory may store instructions that cause the processor to: estimate theFOV of the user by using a sensor included in a wireless earphone,estimate the FOV of the electronic device by using the sensor module,compare the estimated FOV of the user with the estimated FOV of theelectronic device, determine whether the user gazes at the screen of theelectronic device based on the comparison result, recognize the user'sgaze upon determining that the user gazes at the screen of theelectronic device, and perform a specified function based on therecognized user's gaze.

The electronic device according to various embodiments of the disclosuremay further include a communication module configured to establish acommunication connection with the wireless earphone, and the memory mayfurther store instructions that cause the processor to receive the FOVof the user estimated by the wireless earphone through the communicationmodule.

In the electronic device according to various embodiments of thedisclosure, the memory may further store instructions that cause theprocessor to: estimate the posture of the wireless earphone with dataobtained using the sensor included in the wireless earphone, estimatethe posture of the user's head by using the estimated posture of thewireless earphone.

In the electronic device according to various embodiments of thedisclosure, the memory may further store instructions that cause theprocessor to estimate the user's FOV in consideration of the directionof the wireless earphone.

In the electronic device according to various embodiments of thedisclosure, the user's FOV may be determined according to the functionperformed by the electronic device.

In the electronic device according to various embodiments of thedisclosure, the memory may further store instructions that cause theprocessor to: measure the distance to and direction of the wirelessearphone by using UWB signals, and recognize the user's gaze by usingthe measured distance to and direction of the wireless earphone.

In the electronic device according to various embodiments of thedisclosure, the sensor module may include an acceleration sensor and agyro sensor.

In the electronic device according to various embodiments of thedisclosure, the memory may further store instructions that cause theprocessor to: estimate the posture of the electronic device with dataobtained using a sensor included in the electronic device, and estimatethe FOV of the electronic device by using the estimated posture of theelectronic device.

A wireless earphone according to various embodiments of the disclosuremay include: a sensor module, a memory, and a processor, wherein thememory may store instructions that cause the processor to: estimate theposture of the wireless earphone with data obtained using the sensormodule, estimate the posture of the user's head by using the estimatedposture of the wireless earphone, and estimate the user's FOV by usingthe estimated posture of the user's head.

In the wireless earphone according to various embodiments of thedisclosure, the sensor module may include an acceleration sensor and agyro sensor.

An operation method of an electronic device according to variousembodiments of the disclosure may include: estimating the FOV of theuser by using a sensor included in a wireless earphone, estimating theFOV of the electronic device by using a sensor included in theelectronic device, comparing the estimated FOV of the user with theestimated FOV, determining whether the user gazes at the screen of theelectronic device based on the comparison result, recognizing the user'sgaze upon determining that the user gazes at the screen of theelectronic device, and performing a specified function of the electronicdevice based on the recognized user's gaze.

In the operation method of the electronic device according to variousembodiments of the disclosure, estimating the FOV of the user by using asensor included in a wireless earphone may be receiving the FOV of theuser estimated by the wireless earphone.

In the operation method of the electronic device according to variousembodiments of the disclosure, estimating the FOV of the user by using asensor included in a wireless earphone may include: estimating theposture of the wireless earphone with data obtained using the sensorincluded in the wireless earphone, estimating the posture of the user'shead by using the estimated posture of the wireless earphone, andestimating the FOV of the user by using the estimated posture of theuser's head.

In the operation method of the electronic device according to variousembodiments of the disclosure, estimating the FOV of the user by using asensor included in a wireless earphone may be estimating the user's FOVin consideration of the direction of the wireless earphone.

In the operation method of the electronic device according to variousembodiments of the disclosure, the user's FOV may be determinedaccording to the function performed by the electronic device.

In the operation method of the electronic device according to variousembodiments of the disclosure, recognizing the user's gaze may include:measuring the distance to and direction of the wireless earphone byusing UWB signals, and recognizing the user's gaze by using the measureddistance to and direction of the wireless earphone.

In the operation method of the electronic device according to variousembodiments of the disclosure, the sensor included in the electronicdevice may include an acceleration sensor and a gyro sensor.

In the operation method of the electronic device according to variousembodiments of the disclosure, estimating the FOV of the electronicdevice by using a sensor included in the electronic device may include:estimating the posture of the electronic device with data obtained usingthe sensor included in the electronic device, and estimating the FOV ofthe electronic device by using the estimated posture of the electronicdevice.

An operation method of a wireless earphone according to variousembodiments of the disclosure may include: estimating the posture of thewireless earphone with data obtained using a sensor, estimating theposture of the user's head by using the estimated posture of thewireless earphone, and estimating the user's FOV by using the estimatedposture of the user's head.

In the operation method of the wireless earphone according to variousembodiments of the disclosure, the sensor may include an accelerationsensor and a gyro sensor.

The electronic device according to certain embodiments may be one ofvarious types of electronic devices. The electronic devices may include,for example, a portable communication device (e.g., a smart phone), acomputer device, a portable multimedia device, a portable medicaldevice, a camera, a wearable device, or a home appliance. According toan embodiment of the disclosure, the electronic devices are not limitedto those described above.

It should be appreciated that certain embodiments of the presentdisclosure and the terms used therein are not intended to limit thetechnological features set forth herein to particular embodiments andinclude various changes, equivalents, or replacements for acorresponding embodiment. With regard to the description of thedrawings, similar reference numerals may be used to refer to similar orrelated elements. It is to be understood that a singular form of a nouncorresponding to an item may include one or more of the things, unlessthe relevant context clearly indicates otherwise. As used herein, eachof such phrases as “A or B,” “at least one of A and B,” “at least one ofA or B,” “A, B, or C,” “at least one of A, B, and C,” and “at least oneof A, B, or C,” may include all possible combinations of the itemsenumerated together in a corresponding one of the phrases. As usedherein, such terms as “1st” and “2nd,” or “first” and “second” may beused to simply distinguish a corresponding component from another, anddoes not limit the components in other aspect (e.g., importance ororder). It is to be understood that if an element (e.g., a firstelement) is referred to, with or without the term “operatively” or“communicatively”, as “coupled with,” “coupled to,” “connected with,” or“connected to” another element (e.g., a second element), it means thatthe element may be coupled with the other element directly (e.g.,wiredly), wirelessly, or via a third element.

As used herein, the term “module” may include a unit implemented inhardware, software, or firmware, and may interchangeably be used withother terms, for example, “logic,” “logic block,” “part,” or“circuitry”. A module may be a single integral component, or a minimumunit or part thereof, adapted to perform one or more functions. Forexample, according to an embodiment, the module may be implemented in aform of an application-specific integrated circuit (ASIC).

Certain embodiments as set forth herein may be implemented as software(e.g., the program 140) including one or more instructions that arestored in a storage medium (e.g., internal memory 136 or external memory138) that is readable by a machine (e.g., the electronic device 101).For example, a processor (e.g., the processor 120) of the machine (e.g.,the electronic device 101) may invoke at least one of the one or moreinstructions stored in the storage medium, and execute it, with orwithout using one or more other components under the control of theprocessor. This allows the machine to be operated to perform at leastone function according to the at least one instruction invoked. The oneor more instructions may include a code generated by a complier or acode executable by an interpreter. The machine-readable storage mediummay be provided in the form of a non-transitory storage medium. Wherein,the term “non-transitory” simply means that the storage medium is atangible device, and does not include a signal (e.g., an electromagneticwave), but this term does not differentiate between where data issemi-permanently stored in the storage medium and where the data istemporarily stored in the storage medium.

According to an embodiment, a method according to certain embodiments ofthe disclosure may be included and provided in a computer programproduct. The computer program product may be traded as a product betweena seller and a buyer. The computer program product may be distributed inthe form of a machine-readable storage medium (e.g., compact disc readonly memory (CD-ROM)), or be distributed (e.g., downloaded or uploaded)online via an application store (e.g., PlayStore™), or between two userdevices (e.g., smart phones) directly. If distributed online, at leastpart of the computer program product may be temporarily generated or atleast temporarily stored in the machine-readable storage medium, such asmemory of the manufacturer's server, a server of the application store,or a relay server.

According to certain embodiments, each component (e.g., a module or aprogram) of the above-described components may include a single entityor multiple entities, and some of the multiple entities may beseparately disposed in different components. According to certainembodiments, one or more of the above-described components may beomitted, or one or more other components may be added. Alternatively oradditionally, a plurality of components (e.g., modules or programs) maybe integrated into a single component. In such a case, according tocertain embodiments, the integrated component may still perform one ormore functions of each of the plurality of components in the same orsimilar manner as they are performed by a corresponding one of theplurality of components before the integration. According to certainembodiments, operations performed by the module, the program, or anothercomponent may be carried out sequentially, in parallel, repeatedly, orheuristically, or one or more of the operations may be executed in adifferent order or omitted, or one or more other operations may beadded.

What is claimed is:
 1. An electronic device comprising: a sensor; amemory storing instructions; and a processor configured to execute theinstructions to: estimate a field of view (FOV) of a user by usinganother sensor included in a wireless earphone; estimate a FOV of theelectronic device by using the sensor; compare the FOV of the user withthe FOV of the electronic device to obtain a comparison result;determine whether the user gazes at a screen of the electronic devicebased on the comparison result; recognize a gaze of the user based ondetermining that the user gazes at the screen of the electronic device;and perform a specified function based on the gaze of the user.
 2. Theelectronic device of claim 1, further comprising a communicationinterface configured to establish a communication connection with thewireless earphone, wherein the processor is further configured toexecute the instructions to receive the FOV of the user estimated by thewireless earphone through the communication interface.
 3. The electronicdevice of claim 1, wherein the processor is further configured toexecute the instructions to: estimate a posture of the wireless earphonebased on data obtained using the other sensor included in the wirelessearphone; and estimate a posture of a head of the user based on theposture of the wireless earphone.
 4. The electronic device of claim 1,wherein the processor is further configured to execute the instructionsto estimate the FOV of the user based on a direction of the wirelessearphone.
 5. The electronic device of claim 1, wherein the FOV of theuser is determined according to a function performed by the electronicdevice.
 6. The electronic device of claim 1, wherein the processor isfurther configured to execute the instructions to: measure, by usingultra-wideband (UWB) signals, a distance to the wireless earphone and adirection of the wireless earphone; and recognize the gaze of the userbased on the distance to the wireless earphone and the direction of thewireless earphone.
 7. The electronic device of claim 1, wherein thesensor comprises an acceleration sensor and a gyro sensor.
 8. Theelectronic device of claim 1, wherein the processor is furtherconfigured to execute the instructions to: estimate a posture of theelectronic device based on data obtained using the sensor included inthe electronic device; and estimate the FOV of the electronic devicebased on the posture of the electronic device.
 9. A wireless earphonecomprising: a sensor; a memory storing instructions; and a processorconfigured to execute the instructions to: estimate a posture of thewireless earphone based on data obtained using the sensor; estimate aposture of a head of a user based on the posture of the wirelessearphone; and estimate a field of view (FOV) of the user based on theposture of the head of the user.
 10. The wireless earphone of claim 9,wherein the sensor comprises an acceleration sensor and a gyro sensor.11. An operation method of an electronic device, the operation methodcomprising: estimating a field of view (FOV) of a user by using a firstsensor included in a wireless earphone; estimating a FOV of theelectronic device by using a second sensor included in the electronicdevice; comparing the estimated FOV of the user with the estimated FOVof the electronic device; determining whether the user gazes at a screenof the electronic device based on the comparing; recognizing a gaze ofthe user based on determining that the user gazes at the screen of theelectronic device; and performing a specified function of the electronicdevice based on the gaze of the user.
 12. The operation method of claim11, wherein the estimating the FOV of the user comprises receiving, fromthe wireless earphone, the FOV of the user estimated by the wirelessearphone.
 13. The operation method of claim 11, wherein the estimatingthe FOV of the user comprises: estimating a posture of the wirelessearphone based on data obtained using the first sensor included in thewireless earphone; estimating a posture of a head of the user based onthe posture of the wireless earphone; and estimating the FOV of the userbased on the posture of the head of the user.
 14. The operation methodof claim 11, wherein the estimating the FOV of the user comprisesestimating the FOV of the user based on a direction of the wirelessearphone.
 15. The operation method of claim 11, wherein the FOV of theuser is determined according to a function performed by the electronicdevice.
 16. The operation method of claim 11, wherein the recognizingthe gaze of the user comprises: measuring, by using ultra-wideband (UWB)signals, a distance to the wireless earphone and a direction of thewireless earphone; and recognizing the gaze of the user based on thedistance to the wireless earphone and the direction of the wirelessearphone.
 17. The operation method of claim 11, wherein the secondsensor comprises an acceleration sensor and a gyro sensor.
 18. Theoperation method of claim 11, wherein the estimating the FOV of theelectronic device comprises: estimating a posture of the electronicdevice based on data obtained using the second sensor included in theelectronic device; and estimating the FOV of the electronic device basedon the posture of the electronic device.
 19. An operation method of awireless earphone, the operation method comprising: estimating a postureof the wireless earphone based on data obtained using a sensor;estimating a posture of a head of a user based on the posture of thewireless earphone; and estimating a field of view (FOV) of the userbased on the posture of the head of the user.
 20. The operation methodof claim 19, wherein the sensor comprises an acceleration sensor and agyro sensor.